Display system, mobile information unit, wearable terminal and information display method

ABSTRACT

A display system, that is capable of displaying substantially the same content on a wearable terminal that is attached to the head of a user and has a first display section, and on a unit other than the wearable terminal that has a second display, comprises a controller that determines an object the user is gazing at, and a display controller that controls update of display states based on the object that has been determined, when an image, that has been displayed on the second display, is displayed on the first display.

CROSS-REFERENCE TO RELATED APPLICATIONS

Benefit is claimed, under 35 U.S.C. § 119, to the filing date of priorJapanese Patent Application No. 2016-171709 filed on Sep. 2, 2016. Thisapplication is expressly incorporated herein by reference. The scope ofthe present invention is not limited to any requirements of the specificembodiments described in the application.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to a wearable terminal having a glassestype viewfinder display section, a display system, and mobileinformation unit, and to an information display method.

2. Description of the Related Art

In recent years, so-called augmented reality (AR) has been developed,and it has become possible to display images of virtual objects, such asCG (computer graphics) generated by a computer such that the virtualobject is in reality space. For example, there has been proposed awearable terminal whereby the user fits glasses having a display section(the wearable terminal), and the wearable terminal displays variousinformation on a display section superimposed on a reality space if theuser visualizes the reality space by means of this wearable terminal(refer to Japanese patent laid open No. 2011-253324 (hereafter referredto as “patent publication 1”)).

SUMMARY OF THE INVENTION

A display system of a first aspect of the present invention is a displaysystem that is capable of displaying substantially the same content on awearable terminal that is attached to the head of a user and has a firstdisplay, and on a unit other than the wearable terminal that has asecond display, this display system comprising: a controller thatdetermines an object the user is gazing at, and a display controllerthat controls update of display states based on the object that has beendetermined, when an image, that has been displayed on the seconddisplay, is displayed on the first display.

A mobile information unit of a second aspect of the present invention iscapable of communication with a wearable terminal that has a firstdisplay that is attached to the head of a user, and that is capable ofpassing visible light in a direction in which the user directly viewsreality space, the mobile information unit comprising: a second displaythat is provided on the mobile information unit, a controller thatdetects an object the user is gazing at, and a display controller thatcontrols change of display format based on the object, when an image,that has been displayed on the second display, is displayed on the firstdisplay.

A wearable terminal of a third aspect of the present invention iscapable of communication with a mobile information terminal having asecond display and a controller that determines an object that a user isgazing at, a first display provided on the wearable terminal, and adisplay controller that controls change of display format based on theobject, when an image, that has been displayed on the second display, isdisplayed on the first display.

An information display method of a fourth aspect of the presentinvention is capable of displaying substantially the same content on awearable terminal that is attached to the head of a user and has a firstdisplay, and on a unit other than the wearable terminal and that has asecond display, the information display method comprising: detecting anobject by the user is gazing at, and controlling change of displayformat based on the object, when an image, that has been displayed onthe second display, is displayed on the first display.

An information display method of a fifth aspect of the present inventionis an information display method for a mobile information unit that hasa second display, and that is capable of communication with a wearableterminal that has a first display that is attached to the head of auser, and that is capable of passing visible light in a direction inwhich the user directly views reality space, the information displaymethod comprising: detecting an object the user is gazing at, andcontrolling change of display format based on the object, when an image,that has been displayed on the second display, is displayed on the firstdisplay.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A-FIG. 1C are block diagrams mainly showing the electricalstructure of a display system of a first embodiment of the presentinvention.

FIG. 2 is a drawing showing a usage state of the display system of thefirst embodiment of the present invention.

FIG. 3A and FIG. 3B are drawings showing a usage state of the displaysystem of the first embodiment of the present invention.

FIG. 4A and FIG. 4B are flowcharts showing operation of the displaysystem of the first embodiment of the present invention.

FIG. 5 is a flowchart showing operation of mobile terminal control of aninformation display device of a first embodiment of the presentinvention.

FIG. 6 is a flowchart showing operation of a mobile terminal of a firstembodiment of the present invention.

FIG. 7 is a flowchart showing operation of a drone of a first embodimentof the present invention.

FIG. 8 is a flowchart showing operation of a glasses-type terminal of afirst embodiment of the present invention.

FIG. 9A-FIG. 9C are drawings showing usage states of an informationdisplay device of a second embodiment of the present invention.

FIG. 10 is a flowchart showing operation of an in-hospital terminal of asecond embodiment of the present invention.

FIG. 11 is a flowchart showing operation of a wearable section of thesecond embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

A display system comprising a glasses type information display device(hereafter called a wearable section 10) and an external unit 20 will bedescribed in the following as a preferred embodiment of the presentinvention.

FIG. 1A is a block diagram showing the outline of an information displaysystem, and FIG. 1B shows a state where a user 40 has attached thewearable section 10 to their head. The wearable section 10 has aso-called glasses shape, and has optical members 41 and 52, as well asan attachment section for attaching to the head of the user 40. Itshould be noted that the optical members 41 and 42 may be transparentglass or transparent plastic etc. that does not have refractivity. Also,the wearable section 10 has a control section 11, a first displaysection 12 and an information acquisition section 13.

This wearable section 10 functions as a wearable terminal that isattached to the head of a user and has a first display section. Thewearable section 10 also functions as a wearable terminal that isattached to the head of a user and has a first display section that iscapable of passing visible light in a direction in which the userdirectly views a reality space.

The first display section 12 has a display, and when the user 40observes the outside world by means of a right eye optical member 41 animage that has been generated by the control section 11 is displayed onthe display superimposed on a scene of the outside world. This opticalmember 41 is assumed to have a structure having a magnification functionetc. for the display section, and sections besides the optical member 41are transparent using see through elements. However, this is notlimiting, and a translucent reflecting mirror or enlarged half mirror orthe like may also be used.

As shown in FIG. 1C, the user 40 can observe a left field of view(optical binocular field of view) using the left eye optical member 42,but this optical member 42 may be transparent or the optical member maynot be provided. Also, the right field of view (optical binocular fieldof view) can be observed using the right eye optical member 41, but thisoptical member 41 may also be transparent or not provided, as wasdescribed previously. It is also possible for the user 40 to observe animage, formed using the first display section 12, on the display region12 a below the right eye optical member 41.

It should be noted that while the display region 12 a of the firstdisplay section 12 is provided on the side of the right eye opticalmember 41, this is not limiting, and the display region may also beprovided on the side of the left eye optical member 42. At this time thedisplay section constitutes the entire field of view, but if, forexample, an imaging result in front that has been obtained by aninformation acquisition section 13, that will be described later, isdisplayed in order to keep objects right in front of the user frombecoming invisible, the display section can also function similarly tobeing transparent.

This embodiment is proposed in view of the fact that a user will “gaze”at an object that is in the foreground in the real world. With atendency to observe things using not only a sense of vision using theireyes, but also through the other 5 senses, users carry out actionsbetter to observe objects in the real world more carefully. With thisembodiment, actions to see object on the real world more carefully,namely, detecting conditions to see actual objects that are beyond thetransparent, are called “gaze detection” or “gaze determination”.Specifically, carrying out display switching intuitively, unconsciouslyand in a hands-free manner without special operations by detectingconditions such as gazing at an object that has been displayed virtuallyon a glasses type wearable terminal, and more than this, detecting gazestate of the real world accompanying variation in the users attitude andposture etc. is described as this embodiment.

The information acquisition section 13 has an image input section 13 aand a sound input section 13 b. As shown in FIG. 1B, the image inputsection 13 a has a camera 13 a that includes a photographing lens forforming an image, and an image sensor and image processing circuits etc.for subjecting images to photoelectric conversion. Image data that hasbeen converted by the image input section 13 a is output to the controlsection 11. The sound input section 13 b has a microphone 13 bb forconverting voice to voice data, and the voice data that has beenconverted here is output to the control section 11. It should be notedthat in FIG. 1B the first display section 12, image input section 13 a,camera 13 aa, sound input section 13 b and microphone 13 bb are depictedseparately, but may be constructed integrally with the wearable section10.

If the user stares at something, their eyes will be concentrating at aspecified posture, and so “gazing” will be determined when a state suchas there being no variation in the image or sound information for agiven time has been detected. Alternatively, “gazing” may be determinedin a case where an image that has been detected is a monitor image,where there is a image of a specified object that has been stored inadvance by learning or a program, or where a sound that has beendetected is the sound of a specified unit or the voice of the user.

With this embodiment, since display on a display section is switched inaccordance with what is currently being looked at, providing theinformation acquisition section 13 in the wearable section 10, withwhich it is easy to determine the behavior, attitude and posture of theuser, is advantageous. As well as detection of an image the user islooking at, variation and transitions etc. of the image being looked atmay also be detected. The information acquisition section 13 and amovement sensor can both be used in gaze determination, but it ispossible to use only either one of them, or a combination of the two. Ineither case, it becomes possible to determine time from when a givencondition is achieved, and characteristics of transition from a givenfirst state to a given second state etc.

Obviously, learning of that person's habits such as the way of utteringspeech and moving their body (determination of whether or the user issatisfied using sound etc., and making training data) may also be used.Such an information acquisition section and movement sensor section maybe provided separately, and fixation determination performed using thatdata, or fixation determination may also be carried out by another unitreceiving that data. That is, it is possible to determine a fixationstate for the user based on time variation characteristics of the user'sposture, or based on continuation of posture for a specified time. Thisposture can be determined using movement itself, or a result of havingmoved, or an image itself or transition of an image.

The control section 11 has a CPU (Central Processing Unit), peripheralcircuits for that CPU, and memory etc. The CPU implements the wearablesection 10 overall by controlling each of the sections within thewearable section 10 in accordance with programs stored in memory. Theperipheral circuits have an electronic compass for detecting whichdirection the wearable section 10 is facing in, an inclination detectionsensor that measures gaze elevation angle, and a movement sensor sectionsuch as gyro that detects movement.

Also, as a peripheral circuit within the control section 11 there is acommunication circuit, that carries out communication with the externalunit 20. As a communication system wireless communication, such as WiFiand Bluetooth (registered trademark), or infrared communication may beused, and in this case a single communication system may be used, orboth communication systems may be used. Also, in the event that bothsystems are used, image communication may be carried out using Wi-Ficommunication utilizing broadcast mode or the like, and communication ofcontrol signals with the external unit may be carried out usingBluetooth (registered trademark).

The external unit 20 may be a unit having a display section and acommunication circuit, such as a desktop PC (Personal Computer), anotebook PC, tablet PC or smartphone etc. With this embodiment,description will be been given for a case where a smartphone has beenadopted as the external unit 20, but units besides a smartphone may alsobe used.

The external unit 20 has a control section 21, a second display section22, and an information acquisition section 23. The external unit 20functions as a unit other than the wearable terminal having a seconddisplay. This unit other than the wearable terminal and the wearableterminal are capable of displaying substantially the same content. Theexternal unit 20 also functions as a mobile information unit having asecond display and a controller that determines an object being gazed atby a user. It should be noted that this controller is not limited tobeing part of a mobile information unit, and may also be arranged on thewearable terminal. In this case, the controller determines an object theuser is gazing at based on at least one of time variable characteristicsof the user's posture, user's posture information, surrounding soundinformation, and image information that has been acquired using theimage sensor.

The second display section 20 has a liquid crystal panel or organic ELpanel etc., and displays images. As display images, in the case wherethe external unit 20 is a smartphone, there are menu screens forsmartphone etc., and in the case of being linked to a drone, as will bedescribed later, there are images etc. that have been transmitted fromthe drone. The information acquisition section 23 is for the user toinput information to the external unit 20, and is, for example, akeyboard or touch panel etc.

The control section 21 has a CPU (Central Processing Unit), peripheralcircuits for that CPU, and memory etc. The CPU implements the externalunit 20 overall by controlling each of the sections within the externalunit 20 in accordance with programs stored in memory. Also, as aperipheral circuit within the control section 21, there is acommunication circuit that carries out communication with the wearablesection 10 and a second external unit etc.

It should be noted that in FIG. 1A the control section 11 is arranged inthe wearable section 10, and the control section 21 is arranged in theexternal unit 20. However, the function of a display control section 30that carries out display control (including switching control) for thefirst display section 12 and the second display section 22 may beassumed by only the control section 11 or may be assumed by only thecontrol section 21. Also, besides this role being assumed by only onecontrol section, these functions may be implemented by the two controlsections 11 and 21 acting cooperatively.

Whether or not the user is looking at the external unit 20 may bedetermined based on the shape of the external unit 20 using the imageinformation that has been acquired by the image input section 13 a, ormay be determined using display content of the external unit 20. Thedetermination may be supplemented by carrying out display of specialcontent (for example, display for a device allowing easy reading such asa bar code or pattern etc.) on the external unit 20, and the specialdisplay may be combined with an emitted light pattern and apronunciation pattern. Taking a look at the external unit 20 may be theuser's intention, wanting to see it, and so may be classified as“gazing”. By determining using time of gazing, a movement pattern etc.at the time of taking a look may be determined with images and movement(variations in posture).

The display control section 30 functions as a controller for determiningan object that the user is gazing at (for example, S1 in FIG. 4A, FIG.4B, and step S67 in FIG. 6). As was described previously, data forimages, sound and movement etc. may be acquired, and this determinationmay be carried out based on characteristics and change characteristicsetc. of this data. It is possible to further increase performance byutilizing learning that uses artificial intelligence, whereby whether ornot the user is satisfied with a determination result is determinedusing the voice and gestures etc. of the user at the time determinationresults have been displayed, and accumulating results such as good orbad for each situation.

The display control section 30 also functions as a display controllerthat controls change in display format based on an object that has beendetermined, when displaying an object, that has been displayed on thesecond display, on the first display (refer, for example, to S3, S5 andS9 in FIG. 4). This display controller may be provided within a wearablesection that is provided integrally with a display, and may also beprovided at the external unit side. The display controller may also beimplemented in cooperation with a control section that is providedwithin these units, and may also be implemented by cooperating withother units.

The above-mentioned controller may acquire posture information of theuser, and perform determination based on this posture information(refer, for example, to S23, S25 and S31 in FIG. 4B). The controller mayalso acquire surrounding sound information, and perform determinationbased on this sound information that has been acquired (refer, forexample, to S27 and S31 in FIG. 4B). The controller may also performdetermination based on image information that has been acquired by animaging section (refer, for example, to S29 and S31 in FIG. 4B).

The controller may also perform determination based on time variablecharacteristics of the user's posture, or continuation of a posture fora given time. This determination may be carried out not only with theimage, sound and posture information itself that has been acquired, butalso taking into consideration duration time of that information, and aprocess until that posture state is achieved. Learning may be done withmultiple combinations. Accordingly, it may be assumed that the imageinput section (imaging section) and sound input section may haveappropriate directivity so as to change direction in which acquisitionof images and sound is possible in accordance with posture and behaviorof the user. Obviously it is also possible to achieve the same functionsusing an image input section and sound input section that havedirectivity control. Specifically, at the time of directly viewing anobject that is in front of the user's face, an object that can be seentransparently is desirably arranged or constructed with the possibilityof being determined based on information that has been acquired.

Next, an example of use of the display system of this embodiment will bedescribed using FIG. 2 and FIG. 3. FIG. 2 shows appearance of a user 40with a wearable section 10 attached to their head, carrying out shootingwhile operating a remotely operable radio controlled aircraft (drone 50)that is capable of being remotely operated using the external unit 20.It should be noted that a smartphone is used as the external unit 20.Also, with this embodiment a drone is being used, but the invention isnot limited to an aircraft, and may also be applied to a robot,equipment that is capable of moving on land (mobile equipment) orequipment capable of moving on water (mobile equipment).

It is desirable for the drone 50 to be operated visually by the user,from an observation point at which safety is secured. The user 40therefore carries out control of the drone 50 by touch operation of atouch panel 22 a that has been provided on the second display section 22of the external unit 20, while looking at the drone 50 through the righteye optical member 41 and the left eye optical member 42 of the wearablesection 10. By placing the optical sections in a transparent state,unnecessary display is done away with, and it is possible to focuseffort on control.

As will be described later, if the user 40 stares at the drone 50, thereis a switch to eyepiece display mode 1 for drone operation and externalunit display mode 1 (refer to eyepiece display mode 1 of S3→S9→S11 inFIG. 4A). This determination as to whether or not the user 40 is gazingat the drone may be carried out using image recognition on images thathave been taken by the image input section 13 a of the wearable section10. For example, if the drone 50 is in an image, it may be determinedthat the user 40 is gazing at the drone 50. Also, in a case where itbecomes no longer possible to see the operating section (displayed onthe second display section 22 of the external unit 20) in an image froma display state, it may be determined that the user 40 is gazing at thedrone 50. Obviously, if it is known that the drone is being operated,what is being looked at, at the time of operation, may be inferred (gazedetermination) using posture and change in posture, such as whether theuser is gradually looking up or gradually looking down, or looking upinto the sky.

If there has been a switch to eyepiece display mode 1 for droneoperation and external unit display mode 1, then an icon 22 b forforward operation of the drone 50, an icon 22 c for backward operation,an icon 22 d for left turn operation and an icon 22 e for right turnoperation are displayed on the display section 22 a of the external unit20.

Operation guide display is also carried out on the display region 12 aof the right eye optical member 41 (eyepiece display). As operationguide display it is possible to display the above described icons 22b-22 e, and to display images that have been photographed by the drone.With the example shown in FIG. 2, since the user 40 is performing atouch operation on the icon 22 b for forward movement, the icon 22 bdisplayed on the display region 12 a is displayed so as to bedistinguishable from other icons.

Also, the drone 50 is mounted with a camera 51, and movie image datathat has been taken by the camera 51 is transmitted to a communicationcircuit of the wearable section 10 by means of the external unit 20 ordirectly from the drone 50. Image data from the drone 50 is displayed,together with the icons 22 b-22 e, on the display region 12 a. In thisway, observation at the drone viewpoint becomes possible, when itbecomes possible to easily confirm what could not previously be seen. Ifoperation and observation can be performed seamlessly, then it ispossible for anyone to observe an object of observation that is locatedat a place where direct viewing is difficult, in detail, and safely. Itis also possible to achieve advanced observation that combinesoperations that are centered on observation where exposure control andfocus control etc. are carried out, and operations that are focused onsafely getting close to a target object.

The display region 12 a is below the right eye optical member 41, as wasdescribed above, and in a case where images from the drone 50 and theicons of the external unit 20 are not displayed on this display region12 a, the external unit 20 can be seen. As a result, display on thedisplay region 12 a is positioned superimposed on a display panel of theexternal unit 20, and the user 40 does not feel uneasy, such as if theywere performing operations while looking directly at the external unit20.

It should be noted that when operating so as to cause the drone 50 torise up the external unit 20 is swung so as to be urged to the front (inthe direction of arrow A in the drawing), while when operating to causethe drone 50 to drop the external unit 20 is swung so as to be urged toan opposite side (in the direction of arrow B in the drawing).

Next, FIG. 3A shows a state where an object being stared at by the user40 has shifted from the drone 50 to the external unit 20, in order tocarry out camera operation on the camera 51. If the object being staredat is replaced by the external unit 20, then as shown in FIG. 3A it ispossible for the user 40 to be able to see the external unit 20 throughthe right eye optical member 41 and the left eye optical member 42. Aswill be described later, if the user 40 stares at the external unit 20there is a switch to eyepiece display mode 2 for external unit operationand external unit display mode 2 (refer to the eyepiece display mode 1of S3→S5→S7 in FIG. 4A).

Because the object being stared at by the user 40 is not facing thedrone 50, then in order to maintain stable flight an instruction isissued from the external unit 20 to the drone 50 to stop movement andcarry out hovering to remain at the same position. Also, the fact thatthe drone 50 is in a movement halted state is displayed on the displayregion 12 a of the right eye optical member 41.

Together with display for carrying out camera operations, such as menusetc., an image 22 f from the drone 50 is also displayed on the displaypanel of the external unit 20. The user 40 carries out cameraoperations, such as focusing, exposure adjustment, instructing stillpicture shooting etc. while looking at this image.

FIG. 3B shows appearance of the user 49 confirming images etc. of otherpeople 43 that have been taken by the drone 50. In this case also, sincethe user 40 is in a state where they are staring at the external unit20, display for the external unit 20 and the display region 12 a isperformed in eyepiece display mode 2 and external unit display mode 2.

In this way, with this embodiment display on the display region 12 a ofthe wearable section 10 and on the external unit 20 is switched inaccordance with the object being stared at by the user 40. This meansthat it is possible to perform required guide display on the displayregion 12 a of the wearable section 10, and to concentrate on operationof the drone 50. Also, the external unit 20 is capable of carrying outappropriate display as an operating device of the drone 50 in accordancewith the object being stared at by the user 40.

Next, operation of this embodiment will be described using theflowcharts shown in FIG. 4 to FIG. 8. The flowchart shown in FIG. 4Ashows main operation of the display system constituted by the wearablesection 10, external unit 20 and drone 50. This flow is executed byCPUs, respectively provided in the control section 11 within thewearable section 10, the control section 12 within the external unit 20and the control section within the drone 50, controlling each of thesections within each unit in accordance with programs that have beenstored in respective internal memories. It should be noted thatoperation of each unit will be described using the flowcharts shown inFIG. 5 to FIG. 8.

If the processing flow shown in FIG. 4A starts, first statedetermination for the glasses-type terminal is carried out (S1). Here,it is determined whether an object being stared at by the user 40 whohas attached the wearable section 10, which is the glasses-typeterminal, is the drone 50 or the external unit 20. Detailed operation ofstate determination for this glasses-type terminal will be describedlater using FIG. 4B, which is a typical example that is easy todescribe.

If the state determination for the glasses-type terminal has beencarried out in step S1, it is determined whether the objected beingstared at is object 1 or object 2 (S3). Here, it is determined whetheror not the object being stared at is the drone 50 (object 1) or is theexternal unit 20 (external unit 20 (object 2) in accordance with thedetermination result of step S1.

If the result of determination in S3 is object 1, namely that the objectbeing stared at by the user is, with the example of this embodiment, thedrone 50, eyepiece display mode 1 is executed (S5) and external unitdisplay mode 1 is executed (S7). In step S5, drone operation guidedisplay such as shown in FIG. 2 is carried out in the eyepiece, that is,on the display region 12 a at the lower side of the right eye opticalmember 41. In step S7, display for operation of the drone 50 is carriedout on the external unit 20 (with this embodiment, a smartphone).Specifically, the icons 22 b-22 e as shown in FIG. 2 are displayed.

On the other hand, if the determination result for step S3 is object 2,namely that the object being stared at by the user is, with the exampleof this embodiment, the external unit 20, eyepiece display mode 2 isexecuted (S9) and external unit display mode 2 is executed (S11). Instep S9 display of the fact that the drone is in a stopped state, suchas shown in FIG. 3A, is carried out in the eyepiece, namely, on thedisplay region 12 a at the lower side of the right eye optical member41. Also, in Step S11, display for camera operation of the drone 50 iscarried out on the external unit 20. Specifically, drone captured liveview images such as shown in FIG. 3A (or taken images), and menu screensand camera operation icons etc. are displayed.

If external unit display mode 1 has been executed in step S7, or ifexternal unit display mode 2 has been executed in step S11, processingreturns to step S1.

In this manner, in the flow for display section control of the system ofthis embodiment, it is determined whether the user 40 is staring atobject 1 (the drone 50 in this example), or is staring at object 2 (theexternal unit 20 with this example), and the display mode of theeyepiece (display region 12 a) and the display mode of the external unit20 (smartphone) are switched depending on the determination result. Thismeans that on the eyepiece (display region 12 a) of the wearable section10 and on the display section of the external unit 20, display isappropriately switched corresponding to usage state of the system by theuser 40.

It should be noted that with this embodiment object 1 has been made thedrone 50 and object 2 has been made the external unit 20, but this isnot limiting and the objects 1 and 2 can be appropriately selected inaccordance with the system to which the present invention is applied.

Next, detailed operation of the glasses-type terminal statedetermination in step S1 will be described using the flowchart shown inFIG. 4B.

If the flow for glasses-type terminal state determination is entered,determination of communication state is first carried out (S21). Here,it is determined whether the wearable section 10 is carrying outcommunication with either unit. With the example shown in FIG. 2 andFIG. 3A and FIG. 3B, the wearable section 10 is capable of communicationwith the external unit 20, and the external unit 20 is capable ofcommunication with the drone 50, and the other party of thiscommunication is determined. The wearable section 10 can determinedcommunication state between the external unit 20 and/or the drone 50using the external unit 20, and it is possible to acquire informationfor these units.

Next, gaze direction determination is carried out (S23). Here, thedirection in which the wearable section 10 is facing is detected usingan electronic compass within the wearable section 10. Gaze elevationangle determination is also carried out (S25). Here, elevation angle atwhich the wearable section 10 is facing is detected using an inclinationsensor, movement sensor etc. within the wearable section 10.

Next, built-in camera image determination is carried out (S27). Here,images are identified using image recognition, based on image data thathas been acquired by the image input section 13 a within the informationacquisition section 13. Built-in microphone sound determination is alsocarried out (S29). Here, sounds that are in the vicinity are identifiedbased on sound data that has been acquired by the sound input section 13b within the information acquisition section 13.

Once the above processing has been carried out, next objectdetermination is carried out with useful information (S31). Here, thedisplay control section 30 determines an object that is being stared atby a user who has attached the wearable section 10, based on informationthat has been acquired from steps S21 and S29. For example, in a casewhere a result of elevation angle determination is looking up into thesky, a result of camera image determination is a drone video, and aresult of microphone sound determination is that a propeller sound isbeing emitted, it is determined that the object being stared at isobject 1 (drone). On the other hand, if the result of elevation angledetermination is looking down on water, and a result of camera imagedetermination is an external unit video, it is determined that theobject being stared at is object 2 (external unit 20). Thisdetermination has been carried out by the display control section 30,but may also be carried out by either the control section 11 within thewearable section 10 or the control section 21 within the external unit20 alone, or may be carried out by the two control sections acting incooperation with each other.

Once object determination has been carried out in step S31, it is nextdetermined whether or not the object being stared at is object 1 (S33).If the result of determination in step S31 is that the object beingstared at is object 1, “object 1” (S35) is set, while if the objectbeing stared at is object 2, “object 2” (S37) is set.

In this way, in the flow for glasses-type terminal state determination,the state of each unit is determined, and based on the result of thisdetermination it is determined whether the user 40 who has attached thewearable section 10 is staring at object 1 (here, the drone 50), or isstaring at object 2 (here, the smartphone). It should be noted thatdetermination in this flow is based on information for steps S21-S29,but as this determination information some steps may be omitted, andother information may also be acquired. Determination information may beappropriately selected in accordance with the system to which thepresent invention is applied.

It should be noted that in the flow shown in FIG. 4, in order to makedescription easier description has been given emphasizing stationarysituational judgment for each state, but as described previouslyjudgment may be respectively based on given time continuation and avariety of characteristics etc. It is possible to gain increasedaccuracy and satisfaction with determination through transition patternsor multiple decisions, and further, decisions may be by allowinglearning of various parameters and images using artificial intelligence.

Next, operation of each unit will be described using the flowchartsshown in FIG. 5 to FIG. 8. The flowchart shown FIG. 5 shows operation ofthe external unit 20. This flow is executed by the CPU in the controlsection 21 of the external unit 20 controlling each section within theexternal unit 20 in accordance with programs that have been stored ininternal memory.

If the flow for mobile terminal control started, first, list display offunctional order icons is carried out (S41). Here, icons for functionsthat can be executed by the external unit 20 are displayed in a listformat on the second display section 22. As a function displayed herethere is “glasses-type linkage”. Glasses-type linkage is a mode ofchanging display format in accordance with state of a smartphone(external unit 20) used by the user 40, as was described using FIG. 2and FIG. 3. It should be noted that for functions other thanglasses-type linkage also, display of a mode that links to anotherexternal unit, such as, for example, microscope link, endoscope link,etc. may also be performed.

Next, it is determined whether or not glasses-type linkage has beenselected (S43). In a case where the user 40 wishes to selectglasses-type linkage, it is possible to select glasses-type linkage bymoving a cursor using an operation member, or by a touch operation etc.In this step determination is based on operating state of the user 40.

If the result of determination in step S43 is not glasses-type linkage,icons for lower order functions are displayed (S45). Here, for afunction corresponding to an icon that was selected by the user 40,among the icons and that were displayed in step S41, icons for lowerorder functions are displayed on the second display section 22.

Next it is determined whether or not an icon has been selected (S47).Here it is determined whether or not the user 40 has selected from amongthe icons that were displayed in S47. If the result of determination isthat an icon has been selected, a selection function is launched (S49).It is then determined whether or not there is termination (S51). Here itis determined whether or not the function that has been selected hasbeen terminated. If the result of this determination is that thefunction has not been terminated the selected function is executed.

If the result of determination in step S47 is that an icon has not beenselected, it is determined whether or not to return (S53). An icon oroperation member for returning to the initial state (position) isprovided on the external unit 20. In this step, determination is basedon whether this icon has been touched, or the operation member has beenoperated. If the result of this determination is that return has notbeen selected, processing returns to step S45, and lower order functionicons are displayed.

If the result of determination in step S53 is that return has beenselected, or if the result of determination in step S51 is that thefunction has been terminated, processing returns to S41.

Processing returns to previously described step S43, and if the resultof determination in this step is glasses-type linkage, list display ofother linkage functions is carried out (S61 in FIG. 6). Here, the drone50 which is a unit to be linked in the glasses-type linkage, and modesof linking with other external units besides the drone 50, such as, forexample, microscope linkage, endoscope linkage, etc. are displayed.

Next, it is determined whether or not there is drone linkage (S63). In acase where the user 40 wishes to select drone linkage, it is possible toselect drone linkage by moving a cursor using an operation member, or bya touch operation etc. In this step determination is based operationcarried out by the user 40 on the external unit 20. If the result ofthis determination is that the user has selected linkage to other thanthe drone, another function is executed.

If the result of determination in step S63 is that drone linkage hasbeen selected, next a menu for drone linkage displayed (S65). Here,various menus are displayed on the second display section 22. Forexample, in a case where there are a plurality of drones, menus forvarious settings, such as which drone to link to and a communicationsystem between the external unit 20 and the drone 50, initial settingsof a camera mounted on the drone 50, initial settings of the drone 50itself etc. are displayed. If any menu item is selected execution isperformed in line with the selected item.

It is next determined whether or not the subject of observation is theoperating section or the drone (S67). Here it is determined, based onthe glasses-type terminal state determination etc. that was described inFIG. 4B, and whether the object being stared at by the user 40 is thedrone 50, or is the external unit 20 (smartphone) that has an operationsection. With this embodiment, this determination is carried out by thedisplay control section 30. The control section 21 of the external unit20 within the display control section 30 carries out the determinationindependently. Besides this approach, a determination may be carried outby the external unit 20 and the wearable section 10 acting incooperation, and a result that has been determined may be received bythe wearable section 10.

If the result of determination in step S67 is that the subject ofobservation is the drone, operation section guide display is displayedon the eyepiece and the terminal display section (S69). Here, as wasdescribed using FIG. 2, the user 40 is in the state of staring at thedrone 50. Therefore, display is carried out on the display region 12 aof the wearable section 10 and on the external unit 20 (smartphone), sothat the user 40 can concentrate on operation of the drone 50.Specifically, a signal is transmitted to the wearable section 10 so asto display operation icons 22 b-22 e on the display region 12 a of thewearable section 10. Also, the operation icons 22 b-22 e are displayedenlarged on the second display section 22 of the external unit 20, andit is easy for the user 40 to perform operations using a touch operationeven when groping around in conditions where they are not looking at thescreen.

Next, drone operation signals are transmitted in accordance withoperations (S71). Here, operation signals for moving the drone 50forwards, backwards, turning laterally, moving upwards or movingdownwards etc. are transmitted to the drone 50, based on touch state ofthe icons 22 b-22 e and how the external unit 20 is moved by the user(refer to S95 in FIG. 7).

It is next determined whether or not to carry out live view (S73). Thenmay be cases where the user 40 wishes to carry out operation of thedrone 50 while looking at images from the camera 51 that is mounted onthe drone 50. On the other hand the power supply of the wearable section10 is often small, and in cases where it is desired to prevent powersupply consumption the user 40 will not want image display from thecamera 51. Icons or operating sections for live view display areprovided so that it is possible for the user 40 to select whether or notto carry out live view display, and so in this step it is determinedwhether or not to carry out live view based on the operating states ofthese icons or operating sections.

If the result of determination in step S73 is live view, then the camera51 of the drone 50 is started up, and images are displayed on theeyepiece (display region 12 a) (S75). Live view display is carried outon the display region 12 a, as shown in FIG. 3A (refer to S99 in FIG. 7,which will be described later).

On the other hand, if the result of determination in step S73 is to notcarry out live view, the drone camera is turned off and display is alsoturned off (S77). Here, in order to prevent power supply consumption,the camera that is mounted on the drone 50 is turned off, and display ofthe display region 12 a of the wearable section 10 is also turned off.

Returning to S67, if the result of determination in this step is thatthe subject of observation is the operating section, the drone isstopped (S79). As was described previously, when operating the drone 50,from the viewpoint of maintaining safety it is desirable for the user 40to carry out operations in a state of looking directly at the drone 50.For this reason, it is not desirable fly the drone 50 in a state wherethe user 40 is looking at the operation screen (icons 22 b-22 e etc.) ofthe second display section 22 of the external unit 20 (smartphone). Withthis embodiment, therefore, in a case where the subject of observationis the operating section, the external unit 20 instructs the drone 50 toenter a state of being stopped overhead, that is, to perform hovering.

Next, drone stopped display is carried out on the eyepiece (S81). Theuser 40 observes the operating section through the optical member 41 andthe optical member 42 of the wearable section 10. Therefore, there isinstruction by means of communication so as to display the fact that thedrone is in a stopped state on the display region 12 a of the wearablesection 10, as shown in FIG. 3A.

Once the drone stop display has been carried out, it is next determinedwhether or not to carry out live view (S83). Here, similarly to stepS73, the user 40 selects whether or not to carry out live view display,and so determination is based on this selection result.

If the result of determination in step S83 is live view, then the cameraof the drone 50 is started up, and images are displayed on the eyepiece(display region 12 a) (S85). Here, similarly to S75, live view iscarried out on the display region 12 a, as shown in FIG. 3A (refer, toS99 in FIG. 7, which will be described later).

Next, live view is also carried out on the terminal display section(S87). Here, live view display is also carried out on the second displaysection 22 of the external unit 20. Because the user 40 is looking atthe second display section 22 of the external unit 20, live view displayis also carried out on this display section, and it is possible toconfirm the state of the drone 50.

Next, camera control is carried out in accordance with operations (S89).Menu screens for operation of the camera 51, and various icons, such asshooting instruction for still pictures and movies, focusinginstruction, exposure correction instruction, optical filter insertioninstruction, image processing instruction etc. are displayed on thesecond display section 22 of the external unit 20. In this step cameraoperation is carried out in accordance with such operations etc. by theuser 40 (refer to S103 in FIG. 7, which will described later). It shouldbe noted that besides icons, operating members for camera operation maybe provided, and camera control carried out in accordance with operatingstates of these operating members.

Returning to step S83, if the result of determination in this step is tonot carry out live view, the drone camera is turned off and display isalso turned off (S91). Here, in order to prevent power supplyconsumption, the camera that is mounted on the drone 50 is turned off,and display of the display region 12 a of the wearable section 10 isalso turned off.

If the respective processing has been executed in steps S75, S77, S89,and S91, it is next determined whether or not to terminate droneoperation (S93). In the event that the user 40 terminates operation ofthe drone 50, a terminate icon or terminate operation member etc. isoperated. In this step determination is in accordance with theseoperations. If the result of determination in step S93 is that droneoperation will not be terminated, step S67 is returned to.

On the other hand, if the result of determination in step S93 istermination of drone operation, the drone is caused to return home(S95). Here control is carried out so as to return home, such that thedrone 50 returns safely to where the user 40 is (refer to S95 in FIG. 7,which will be described later). Once the drone 50 returns home, step S41is returned to.

In this way, in the flow for mobile terminal control, if glasses-typelinkage or drone linkage are selected (refer to S43 and S63), display onthe display region 12 a of the wearable section 10 and on the seconddisplay section 22 of the external unit 20 is switched in accordancewith a subject of observation, namely whether the user 40 is staring atthe drone 50 or is staring at the external unit 20 (S69, S81). As aresult, appropriate display is performed in accordance with the objectbeing stared at by the user. Also, since the wearable section 10 islinked to the external unit 20, information possessed by the externalunit 20, for example, operating process and history information of thedrone 50 can be utilized, making comprehensive judgment possible.Further, there is linkage to external devices and databases by means ofthe external unit 20, and it is possible to utilize artificialintelligence.

It should be noted that this flow is carrying out control of the drone50 with the mobile terminal (smartphone). However, this is not limiting,and it is also possible to carry out control using a dedicatedcontroller. In this case also, control of the drone 59 may be carriedout using the glasses-type terminal (wearable section). Also, in theflow shown in FIG. 5 and FIG. 6, in order to make description easierdescription has been given emphasizing situational determination foreach state, but as described previously determination may berespectively based on given time continuation and a variety ofcharacteristics etc. It is possible to gain increased accuracy andsatisfaction with determination through transition patterns or multipledecisions, and further, decisions may be by allowing learning of variousparameters and images using artificial intelligence.

Next, operation of the drone 50 will be described using the flowchartshown in FIG. 7. This flow is executed by the CPU in the control sectionof the drone 50 controlling each section within the drone 50 inaccordance with programs that have been stored in internal memory.

If the flow for drone control is started, first communication is awaited(S91). With this embodiment, the external unit 20 has a role as anoperation unit for the drone 50, and the drone 50 operates on the basisof control signals from the external unit 20. In this step transmissionof signals from the external unit 20 to the drone 50 is awaited. Ifthere is communication, steps S93 and after are executed.

Next, it is determined whether or not it is movement operationcommunication (S93). In a case where the user 40 is staring at the drone50, drone operation signals will be transmitted from the external unit20 (refer to S71 in FIG. 6). In this step it is determined whether ornot communication from the external unit 20 is communication formovement operation.

If the result of determination in step S93 is that it is movementoperation communication, movement control in accordance with operationis carried out (S95). Here, control such as moving forwards, movingbackwards, left turn, right turn, going up, going down and hovering etc.are carried out in accordance with instructions from the external unit20. Also, in the event that a return home instruction is issued from theexternal unit 20, the drone returns to the original position of the user40.

If the result of determination in step S93 is that it is not movementoperation communication, it is determined whether or not it is live viewcommunication (S97). As was described previously, in a case where theuser 40 wishes to confirm live view images that have been taken by thecamera section of the drone 50, the drone camera is started up and aninstruction is issued so as to transmit images. In this stepdetermination is based on whether or not this instruction has beenreceived.

If the result of determination in step S97 is that it is live viewcommunication, images are sent to the unit designated for transmission(S99). Here, the camera 51 that is mounted on the drone 50 is startedup, and image data that has been acquired by this camera 51 istransmitted to the unit that issued the instruction. As the units to bedesignated, there is the wearable section 10 and/or the external unit20, and other units may also be designated for transmission.

If the result of determination in step S97 is that it is not live viewcommunication, it is next determined whether or not there is camera etc.(S101). In a case where the user 40 carries out operation of the camera51 that has been mounted on the drone 50, operating instructions aretransmitted (refer to S89 in FIG. 6). In this step, it is determinedwhether or not such an instruction has been issued.

If the result of determination in step S101 is camera mode etc.,shooting control in accordance with operation is carried out (S103).Here, shooting control is carried out in accordance with instructionsfrom the external unit 20.

If the result of determination in step S101 is not camera mode etc., andif processing of steps S95, S99 and S103 has been executed, processingreturns to step S91.

In this way, in the drone control flow of this embodiment, control ofthe drone 50 is carried out in accordance with commands that have beeninstructed by communication through the external unit 20 by the user 40.In the drone 50 also, in a case where it is desired to prevent powersupply consumption, shooting operations and transmission of taken imagesetc. may be restricted. Also, in the flow shown in FIG. 7, in order tomake description easier description has been given emphasizingstationary state-like situational determination for each state, but asdescribed previously determination may be respectively based on giventime continuation and a variety of characteristics etc. It is possibleto gain increased accuracy and satisfaction with determination throughtransition patterns or multiple decisions, and further, decisions may beby allowing learning of various parameters and images using artificialintelligence.

Next, operation of the glasses-type terminal (wearable section 10) willbe described using the flowchart shown FIG. 8. This flow is executed bythe CPU in the control section 11 of the wearable section 10 controllingeach section within the wearable section 10 in accordance with programsthat have been stored in internal memory.

If the flow for glasses-type terminal control is started, first,communication is awaited (S111). With this embodiment, the wearablesection 10 is operated on the basis of communication from the externalunit 20. In this step transmission of signals from the external unit 20to the wearable section 100 is awaited.

It is next determined whether or not there is communication (S113). Aswas described previously, as a communication system any one or aplurality of wireless communication, such as Wi-Fi or Bluetooth(registered trademark), or infrared communication is used. In this stepit is determined whether or not there has been access from the externalunit 20 using either communications system.

If the result of determination in step S113 is that there iscommunication, each sensor is turned on (S115). As was describedearlier, with this embodiment, the wearable section 10 has varioussensors such as an electronic compass for gaze direction detection, asensor for gaze elevation angle detection, a built-in camera, a built-inmicrophone etc. (refer to FIG. 4B), and operation of these sensors iscommenced.

If the various senses have been turned on, next determination of anobject being stared at is carried out (S117). Here, determination of theobject being stared at is carried out based on the flow of FIG. 4B etc.,using detection outputs of the various sensors whose operation wasstarted step S115.

If determination of object being stared at has been carried out, nextthe determination result is transmitted (S119). Here, the determinationresult of the object being stared at of step S117 is transmitted to theexternal unit 20 by means of a communication circuit. The external unit20 carries out determination of object being stared at based on thisdetection result, and based on this determination result switching ofdisplay is carried out (refer to S69 and S81 in FIG. 6).

Next, it is determined whether or not it is command communication(S121). Here it determined whether or not a command has been transmittedfrom the external unit 20 (smartphone) to the glasses-type terminal top

If the result of determination in step S121 is that a command has beentransmitted, control (display) in line with the command is carried out(S123). As was described previously, since the external unit 20transmits commands for display control to the wearable section 10 (S69,S75, S77, S81, S85, S87 etc. in FIG. 6), control is carried out inaccordance with these commands. For example, as was describedpreviously, in the case where it is desired o prevent power consumptionof the wearable section 10, a command to turn live view display off istransmitted. In this case, live view display is not carried out on thewearable section 10.

If control in line with the command has been carried out step S123, orif the result of determination in step S121 was not commandcommunication, or if the result of determination in step S113 was notcommunication, step S111 is returned to.

In this way, in the flow for the glasses-type terminal, display iscarried out in accordance with commands from the external unit 20 (referto S123). As a result appropriate display in accordance with conditionsat the glasses-type terminal and the external unit becomes possible. Itis also possible to prevent power supply consumption by turning liveview display off.

It should be noted that in the flow shown in FIG. 8, in order to makedescription easier description has been given emphasizing stationarystate-like situational determination for each state, but as describedpreviously determination may be respectively based on given timecontinuation and a variety of characteristics etc. It is possible togain increased accuracy and satisfaction with determination throughtransition patterns or multiple decisions, and further, decisions may beby allowing learning of various parameters and images using artificialintelligence.

As has been described above, the first embodiment of the presentinvention is a system comprising the wearable section 10 applied to aglasses-type terminal, the external unit 20 applied to a smartphone, andthe drone 50, and operation is made easy by making content that isdisplayed on the wearable section 10 and the external unit 20 different,depending on the object being stared at by the user 40.

Next, a second embodiment of the present invention will be describedusing FIG. 9A to FIG. 11. With the second embodiment, when a doctor 44who has attached the wearable section 10 examines a patient, it ispossible to display various information on a display section.

Regarding the structure of this embodiment, the wearable section 10 ofFIG. 1A is worn by a doctor 44, with a personal computer (PC) 60 beingused as the external unit 20, and the main structure of these componentsis the same as in the first embodiment. It should be noted that adisplay panel of a personal computer is suitable for the second displaysection 22 of the external unit 20, and the second display section 22has a control section 21 and an information acquisition section 23.

As the information acquisition section 23 there are a keyboard, touchpanel, microphone etc. As a peripheral circuit of the control section 21there is a communication circuit for carrying out communication with thewearable section 10. The personal computer 60 may be a stand-alonecomputer, but with this embodiment will be described as a hospitalterminal connected to a LAN (local area network) within a hospital.

Next, a method of using the wearable section 10 and personal computer 60of this embodiment will be described using FIG. 9A-FIG. 9C.

FIG. 9A shows the appearance of a doctor 44 who has attached thewearable section 10 examining a patient 45. In this state, the doctor 44who has attached the wearable section 10 is staring at the patient 45.Images that have been acquired by the image input section 13 a providedin the wearable section 10 are only displayed on a display area 12 a ofthe first display section 12 of the wearable section 10, without beingdisplayed on a display monitor of the personal computer 60. Also, liveview images at this time are transmitted to the personal computer 60 andstored in memory within the personal computer 60.

FIG. 9B shows the appearance when the doctor 44 is explaining symptoms(result of diagnosis) to the patient 45. In this state, the doctor 44who has attached the wearable section 10 is staring at the displaymonitor of the personal computer 60. The display region 12 a of thewearable section 10 is turned off, and images of the patient that havebeen stored in memory are displayed on the display monitor of thepersonal computer 60.

FIG. 9C shows a modified example of display on the display panel of thepersonal computer 60. Specifically, with the example shown in FIG. 9B ifthe doctor 44 stares at the patient 45, affected parts of the patient 45are automatically displayed on the display panel of the personalcomputer 60. By contrast, with the modified example shown in FIG. 9C,the patient 45 asks “let me see”, and if in reply to this the doctor 44answers “yes”, that is, if permission is granted in conversation,affected parts of the patient 45 are displayed on the display panel ofthe personal computer 60. As a result, the personal computer 60 performsdetermination by carrying out speech recognition using speech data thathas been collected by a microphone. If display is performed on thedisplay panel of the personal computer 60, display for the displayregion 12 a of the wearable section 10 may be turned off.

It should be noted that affected parts of the patient 45 may bedisplayed in parallel on the display region 12 a, as shown in FIG. 9C.In this case previously taken and stored images are displayed for thepurpose of comparison with images that have been taken during thecurrent diagnosis. In FIG. 9C, after permitting display by conversation,displaying two images in parallel on the display panel of the personalcomputer 60 is for the purpose of comparing previous and current images.

Next, operation of the personal computer 60 will be described using theflowchart shown in FIG. 10. This flow is performed by a CPU of a controlsection within the personal computer 60 controlling each section withinthe personal computer 60 in accordance with programs that have beenstored in an internal memory.

If the flow for the hospital terminal shown in FIG. 10 is started,first, operation is awaited (S131). For example, it is determinedwhether or not various operations have been performed, such as with amouse, a keyboard, or a barcode reader, and if there are none of theseoperations a standby state is entered. If the result of determination instep S131 is that operation has been detected, terminal communicationwith the wearable section 10 is carried out (S133). Here, wirelesscommunication is carried out between the personal computer 60 and thewearable section 10 (refer to S161 in FIG. 11, which will describedlater).

If wearable terminal communication has been carried out, it is nextdetermined whether or not an input operation for the purpose ofinformation acquisition has been performed (S135). As input operationsfor the purpose of information acquisition, there are keyboard input,and sound input using a microphone etc. In this step it is determinedwhether or not any information input has been performed.

If the result of determination in step S135 is that an information inputoperation has been performed, it is next determined what was the inputsource for information acquisition (S137). Here it is determined whetherthe information input source is a keyboard etc., or is speech input. Itshould be noted that when determining the input source for informationacquisition, information of the wearable section 10 may also be used(refer to S163 and S165 in FIG. 11, which will be described later).

If the result of determination in step S137 is sound input, informationis acquired and eyepiece display is carried out (S139). Here, sound datais input from a microphone, and a patient is specified, for example,based on this input. For example, when the doctor 44 designates thepatient's name etc. speech recognition is carried out. Display ofinformation that has been acquired is carried out on the display region12 a of the wearable section 10. For example, in a case where a patienthas been specified, the patient's identity etc. displayed.

Next, speech discussion determination is carried out (S143). Here,content of a conversation between the doctor 44 and the patient 45 isdetermined based on sound data that has been input from the microphone.For example, determination is carried out based on conversation such aswas shown in FIG. 9C. It should be noted that this recognition ofconversation content (speech recognition) may be carried out by thepersonal computer 60, or may be carried out by a server in the hospital,and further, an external server may be connected to and the speechrecognition carried out using this external server.

If the result of determination in step S143 is that content discussed inspeech could be determined (if it has been shared), informationacquisition is performed and desktop display is carried out (S145).Here, information corresponding to the results of speech discussion isrequired, and this information that has been acquired is displayed onthe display panel of the personal computer.

Returning to step S137, if the result of determination in this step isthat there is an input source such as a keyboard, information isacquired and desktop display is carried out (S141). Here, informationthat has been acquired by the keyboard etc. is displayed on the displaypanel of the personal computer 60.

If information has been acquired and desktop display carried out stepS141 or step S145, or if the result of determination in step S143 isthat the result of speech discussion has not been shared, or if aninformation acquisition input has not been performed in step S135,determination of gaze direction is carried out (S147). Here it isdetermined whether the doctor 44 is facing towards the patient 45, or ifthe doctor 44 is facing towards the display panel of the personalcomputer 60. As a determination method for gaze direction, determinationmay be based on conditions shown in the flowchart of FIG. 4B or thelike. It should be noted that when determining gaze direction,information of the wearable section 10 may also be used (refer to S163and S165 in FIG. 11, which will be described later).

If the result of determination in step S147 is that the gaze directionof the doctor 44 is in the direction of the patient 45, eyepiece displayis carried out (S149). Here, an image that was acquired by the imageinput section 13 a of the wearable section 10 is displayed on thedisplay region 12 a of the wearable section 10, as shown in FIG. 9A.

On the other hand, if the result of determination in step S147 is thatthe doctor 44 is facing towards the display panel of the personalcomputer 60, desktop display is carried out (S151). Here, display on thedisplay region 12 a of the wearable section 10 is turned off, anddisplay of affected parts of the patient 45 etc., and information thatis helpful in describing the result of diagnosis to the patient, iscarried out display panel of the personal computer 60.

If display has been carried out in step S149 or step S151, it is nextdetermined whether or not the user has logged out (S153). Here, isdetermined whether or not logout processing has been performed on thepersonal computer 60. If the result of this determination is that logoutprocessing has not been performed, processing returns to step S133.

On the other hand, if the result of determination in step S153 is thatlogout processing has been performed, the wearable terminal is turnedoff (S155). If off processing has been performed for the wearablesection 10 (S177 in FIG. 11) this fact is communicated, anddetermination is based on this communication. If the wearable terminalhas been turned off, step S131 is returned to.

In this way, in the hospital terminal flow, the wearable section 10 canprocess information and appropriately switch display while carrying outcommunication with a hospital terminal such as a PC or the like. Itshould be noted that with this embodiment the determination as towhether or not there will be live view, in steps S73 and S83 in FIG. 6,is not carried out. However, similarly to the first embodiment, liveview display may be turned on or off in order to prevent power supplyconsumption in the wearable section 10. Also, the hospital terminal maybe connected to a hospital server or an external server, and may carryout external processing such as speech recognition and image recognitiontogether with acquiring various information. By making external linkagepossible, various operations are made simple.

Next, operation of the wearable section 10 of this embodiment will bedescribed using the flowchart shown in FIG. 11. This flow is executed bythe CPU in the control section 11 of the wearable section 10 controllingeach section within the wearable section 10 in accordance with programsthat have been stored in internal memory.

If the flow for the wearable section shown in FIG. 11 started, first,communication is awaited (S161). The hospital terminal (personalcomputer 60) carries out communication with the wearable section 10 instep S133, and so this communication is awaited.

If the result of determination in step S161 is that there iscommunication, various input sections are started up (S163). Here,various input sections such as the image input section 13 a and thesound input section 13 b within the wearable section 10 are started up.

If the various intersections have been started up, next various inputresults are transmitted (S165). Here, information that has been acquiredby the input sections that were started up in step S163 is transmittedto the hospital terminal (personal computer 60). It should be noted thatwhen transmitting information, it may not be that data directly.

If transmission of various input results has been carried out, it isnext determined whether the eyepiece (display region 12 a) is capable ofdisplay (S167). In accordance with the previously describeddetermination result for input source for information acquisition, instep S139 display is possible on the eyepiece, while on the other handin step S141 display is not carried out on the eyepiece. Also, dependingon the gaze direction of the doctor 44, display is possible on theeyepiece at step S149, but display is not carried out on the eyepieceS151. In this step, determination as to whether or not the eyepiece iscapable of display is based on instruction from the hospital terminal.

If the result of determination in step S167 is that eyepiece display ispossible, eyepiece display is carried out (S171). Here, display such asshown in FIG. 9A, for example, is carried out on the display region 12a. On the other hand, if the result of determination is that eyepiecedisplay is not possible, eyepiece display is turned off (S169). Here,display on the display region 12 a is turned off as shown, for example,in FIG. 9B. It should be noted that as long as it is not inconvenientdisplay may be maintained as it is. By turning display off, it ispossible to prevent power supply consumption in the wearable section 10.

If eyepiece display has been carried out in step S171, next it isdetermined whether or not a shooting instruction or cancellation hasbeen performed (S173). If a shooting instruction is issued, stillpicture shooting based on image data that has been acquired by the imageinput section 13 a is instructed, and images are displayed until theshooting is cancelled. Here it is determined whether or not thisshooting instruction or cancellation has been carried out. Also, in thecase of a movie, it is determined whether commencement of movie shootinghas been instructed or completion of movie shooting has been instructed.

If the result of determination in step S173 is that shooting instructionor cancellation has been issued storage or cancellation of shootingresults is carried out (S175). Here, display of a still image (freeze)and commencement and completion of movie storage are carried out basedon the result of determination in step S173. It should be noted thatstorage of image data that has been acquired by the image input section13 a is transmitted to the personal computer 60 and carried out in thepersonal computer 60, but storage may also be carried out by thewearable section 10. If shooting results have been canceled, normal liveview display is returned to.

If saving or cancellation of shooting results has been performed in stepS175, or if eyepiece display has been turned off in step S169, or if theresult of determination in step S173 is that there was not instructionor cancellation of shooting, it is next determined whether or not thereis terminal off information (S177). As was described previously, ifturning off of the wearable terminal is carried out in step S155 duringoperation of the hospital terminal, this is notified by means ofcommunication. In this step determination is based on whether or notthere is this notification. If the result of this determination is thatthere is no terminal off information, processing returns to step S161.

If the result of determination in step S177 is that there is terminaloff information, various input sections are turned off (S179). Here, thevarious input devices that were started up in step S163 are turned off.Once the various input devices have been turned off, processing returnsto step S161.

In this way, with the second embodiment of the present invention, in thecase where the doctor 44 is staring at the patient 45, images of theaffected parts of the patient etc. are displayed on the display region12 a of the wearable section 10 (refer to FIG. 9A, S149 in FIG. 10, andS171 in FIG. 11). Also, in a case where the doctor 44 is explainingsymptoms to the patient 45 while looking at the personal computer 60,images are displayed on the personal computer 60 (refer to FIG. 9B, S151in FIG. 10, and S169 in FIG. 11 etc.). As a result, with this embodimentalso, it is possible to make display of display sections of the wearablesection 10 and the external unit 20 (personal computer 60) appropriatein accordance with gaze direction of the user who has fitted thewearable section 10.

Also, with the second embodiment, when the wearable terminal has beenfitted, there are reaction information acquisition sensors (image sensorof the image input section 13, microphone of the sound input section 13b, etc.) for acquiring reaction information of a person opposite, and adisplay controller displays an image, that was displayed on a seconddisplay, on a first display unit, based on reaction information that hasbeen acquired by the reaction information sensor (refer, for example, toFIG. 9C and S143 and S145 in FIG. 10). As a result it is possible toautomatically carry out appropriate display on the first display sectionand the second display section in accordance with the surroundingsituation. It should be noted that as the reaction informationacquisition sensor, with this embodiment, for example, there are themicrophone, electronic compass, image input section, and movementsensors such as an inclination detection sensor and gyro etc. but userreaction information may also be detected with other sensors.

As has been described above, with each of the embodiments of the presentinvention, an object that the user is staring at is detected (refer, forexample, to S1 in FIG. 4A, FIG. 4B, and S67 in FIG. 6 etc.), and whendisplaying an image, that has been displayed by the second displaysection, on the first display section switching is controlled based onthe object (refer, for example, to S3, S5, and S9 in FIG. 4A and S69 andS81 in FIG. 6 etc.). As a result it is possible to appropriately carryout display on the wearable terminal and on an external device inaccordance with usage states of the wearable terminal by the user.Specifically, it is possible to carry out appropriate switching ofdisplay in accordance with an object that the user is gazing at.

It should be noted that in each of the embodiments of the presentinvention description has been given for an example where a radiocontrolled aircraft is controlled by means of a smartphone, and anexample where the present invention has been applied to a case of adoctor diagnosing a patient. However, it is possible to apply thepresent invention to any example as long as it is a system that carriesout display on both a wearable terminal and another external unit.

Also, with each of the embodiments of the present invention, descriptionhas been given for examples where control sections are executed by aCPU, peripheral circuitry for the CPU, and programs. However, this isnot limiting, and it is also possible for some or all functions to havea hardware structure such as gate circuits that have been generatedbased on a programming language that is described using Verilog, andalso to use a hardware structure that utilizes software such as a DSP(digital signal processor). Suitable combinations of these approachesmay also be used. Also, results of learning using artificialintelligence may be reflected in various detection functions.

Also, among the technology that has been described in thisspecification, with respect to control that has been described mainlyusing flowcharts, there are many instances where setting is possibleusing programs, and such programs may be held in a storage medium orstorage section. The manner of storing the programs in the storagemedium or storage section may be to store at the time of manufacture, orby using a distributed storage medium, or they be downloaded via theInternet.

Also, regarding the operation flow in the patent claims, thespecification and the drawings, for the sake of convenience descriptionhas been given using words representing sequence, such as “first” and“next”, but at places where it is not particularly described, this doesnot mean that implementation must be in this order.

As understood by those having ordinary skill in the art, as used in thisapplication, ‘section,’ ‘unit,’ ‘component,’ ‘element,’ ‘module,’‘device,’ ‘member,’ ‘mechanism,’ ‘apparatus,’ ‘machine,’ or ‘system’ maybe implemented as circuitry, such as integrated circuits, applicationspecific circuits (“ASICs”), field programmable logic arrays (“FPLAs”),etc., and/or software implemented on a processor, such as amicroprocessor.

The present invention is not limited to these embodiments, andstructural elements may be modified in actual implementation within thescope of the gist of the embodiments. It is also possible form variousinventions by suitably combining the plurality structural elementsdisclosed in the above described embodiments. For example, it ispossible to omit some of the structural elements shown in theembodiments. It is also possible to suitably combine structural elementsfrom different embodiments.

What is claimed is:
 1. A display system that is capable of displayingsubstantially the same content on a wearable terminal that is attachedto the head of a user and has a first display, and on a unit other thanthe wearable terminal and that has a second display, comprising: acontroller that determines an object the user is gazing at, and adisplay controller that controls change of display format based on theobject that has been determined, when an image, that has been displayedon the second display, is displayed on the first display.
 2. The displaysystem of claim 1, wherein: the controller performs determination basedon time variable characteristics of the user's posture, or continuationof a posture for a given time.
 3. The display system of claim 1,wherein: the controller acquires posture information of the user, andperforms determination based on the posture information.
 4. The displaysystem of claim 1, wherein: the controller acquires surrounding soundinformation, and performs determination based on this sound informationthat has been acquired.
 5. The display system of claim 1, wherein: thecontroller performs determination based on image information that hasbeen acquired by an image sensor.
 6. The display system of claim 1,further comprising: a reaction information acquisition sensor thatacquires reaction information of a person opposite, when the wearableterminal has been fitted, and the display controller displays an image,that has been displayed on the second display, on the first display,based on the reaction information that has been acquired by the reactioninformation position sensor.
 7. A mobile information unit that iscapable of communication with a wearable terminal that has a firstdisplay that is attached to the head of a user, and that is capable ofpassing visible light in a direction in which the user directly viewsreality space, comprising: a second display that is provided on themobile information unit, a controller that detects an object that theuser is gazing at, and a display controller that controls change ofdisplay format based on the object, when an image, that has beendisplayed on the second display, is displayed on the first display. 8.The mobile information unit of claim 7, wherein: the controller performsdetermination based on time variable characteristics of the user'sposture, or continuation of a posture for a given time.
 9. The mobileinformation unit of claim 7, wherein: the controller acquires postureinformation of the user, and performs determination based on the postureinformation.
 10. The mobile information unit of claim 7, wherein: thecontroller acquires surrounding sound information, and performsdetermination based on this sound information that has been acquired.11. The mobile information unit of claim 7, wherein: the controllerperforms determination based on image information that has been acquiredby an imaging section.
 12. The mobile information unit of claim 7,further comprising: a reaction information acquisition sensor thatacquires reaction information of a person opposite, when the wearableterminal has been fitted, and wherein the display controller displays animage, that has been displayed on the second display, on the firstdisplay, based on the reaction information that has been acquired by thereaction information position sensor.
 13. A wearable terminal that iscapable of communication with a mobile information unit that has seconddisplay and a controller that determines an object that a user is gazingat, comprising: a first display provided on the wearable terminal, and adisplay controller that controls change of display format based on theobject, when an image, that has been displayed on the second display, isdisplayed on the first display.
 14. The wearable terminal of claim 13,further comprising: a controller that determines an object the user isgazing at, based on at least one of time variable characteristics of theuser's posture, user's posture information, surrounding soundinformation, and image information that has been acquired using an imagesensor.
 15. An information display method that is capable of displayingsubstantially the same content on a wearable terminal that is attachedto the head of a user and has a first display section, and on a unitother than the wearable terminal that has a second display, comprising:detecting an object that the user is gazing at, and controlling changeof display format based on the object, when an image, that has beendisplayed on the second display, is displayed on the first display. 16.The information display method of claim 15, wherein the object the useris gazing at is determined based on at least one of time variablecharacteristics of the user's posture, user's posture information,surrounding sound information, and image information that has beenacquired using an image sensor.
 17. An information display method for amobile information unit that has a second display, and that is capableof communication with a wearable terminal that has a first display andthat is attached to the head of a user, and that is capable of passingvisible light in a direction in which the user directly views realityspace, comprising: detecting an object that the user is gazing at, andcontrolling change of display format based on the object that has beendetected, when an image, that has been displayed on the second display,is displayed on the first display.
 18. The information display method ofclaim 17, wherein: the object the user is gazing at is determined basedon at least one of time variable characteristics of the user's posture,user's posture information, surrounding sound information, and imageinformation that has been acquired using an image sensor.